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INTRODUCTION 


What is learning 
engineering? 
by Jim Goodell 


Many people are uncomfortable when the words learning and engineering are used 
together. Online discussion threads on the topic of learning engineering prompted 
strong reactions. One comment compared learning to the weather—a complex 
outcome of social, cultural, and economic conditions that cannot be engineered (or 
engineered for). Someone else correctly pointed out that learning is what a learner 
does, not something done to the learner. 

Learning is a human process, and “engineering” sounds too mechanical or 
industrial. The “art” of teaching sounds more natural. When talking about creating 
solutions for learning, isn’t “design” a better term? The words learning engineering 
justifiably raise concerns and questions, especially for professionals who care deeply 
about human learning at all levels. In the following pages, we define learning engi- 
neering—it’s engineering for learning. Learning engineers don’t engineer learning 
any more than civil engineers engineer civilians. Just as civil engineers create and 
maintain physical infrastructure for the benefit of civilians, learning engineering 
creates solutions for the benefit of learners and learning. 

People from diverse fields of practice have wondered, is learning engineering 
just a new label on what we already do? Is it possible for a teacher, instructional 
designer, learning scientist, data scientist, or instructional system designer to do 
learning engineering? If it’s different from what people already do, how is it different? 

Through the following pages, we hope to answer those questions and make 
the case for learning engineering as a professional practice that complements related 
professional practices and fields of study. We offer stories about teams of professionals 
doing learning engineering—whether they call it learning engineering or use a 
different name. In practice, however, what most education, training, and learning 
analytics professionals do daily only partially overlaps with the practice of learning 
engineering. Learning engineering is often concerned with solving problems that 
teachers, curriculum designers, and textbook publishers are not usually asked to 
solve. 
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Complex systems, like weather patterns, produce emergent behavior that can be 
anticipated, but not engineered. 


You cannot engineer “learning,” similarly as you cannot engineer “happiness.” 


l also agree with you as “engineering learning” seems to make learning 
mechanical. Learning is a natural process. It can be analyzed, guided, supported 
and much more but “engineering” learning would be killing the natural process 
which is essential for the joy of learning. Learning analytics is more about 
analyzing to understand or improve the process, but not engineering it. Also, 
when we work on individual learning, personalized learning, it’s a natural 
process with unique abilities of the learner, whereas “engineering learning” 
seems to take back to ‘old school’ where all minds were treated alike. 


|, like you, would like to see what’s the thing that learning engineering does 
which is not done within LA/EDM/LS in order to understand the need of a 
separately “branded” community. 


FIGURE 0.1. Discussion thread about learning engineering (adapted from real discussion) 


This book offers learning and e-learning professionals opportunities to im- 
prove their work with additional engineering tools, techniques, and mindsets (with- 
out changing the name of what they do). And it offers foundational knowledge and 
tools for those wishing to practice as learning engineers. 


[1] 


In 1928, Scottish scientist Alexander Fleming discovered penicillin. The discovery 
was a scientific breakthrough that saved countless lives during and since World War 
II. The 1945 Nobel Prize in Physiology or Medicine was awarded to Fleming and 
two Oxford University scientists who made progress in making concentrated peni- 
cillin and proved the effects of the drug. However, the Nobel Prize—winning scien- 
tists never developed the ability to produce the drug at scale. The Oxford team found 
that it was impossible to produce in sufficient quantities in their laboratory. 
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In 1941, mounting deaths from World War II battle wounds prompted these 
British scientists to reach out for help from the United States. The Rockefeller 
Foundation in New York arranged for Howard Florey and Norman G. Heatley to 
travel in July 1941 to meet with Charles Thorn, the US Department of Agriculture’s 
chief mycologist. As a result of that trip, thirty-nine separate drug laboratories be- 
gan producing penicillin. The United States entered the war later that year when the 
Japanese bombed the American fleet in Pearl Harbor, Hawaii. 

But by June 1942, the US labs had only enough penicillin available to treat 
about ten patients. The urgency of lives being lost in the war meant that production 
of penicillin needed to move out of the laboratory and into mass-production. This 
was no longer just a scientific endeavor; it required engineering. 

The goals of science and engineering are different. The goal of science is 
to discover the truth about the world as it is. The goal of engineering is to create 
scalable solutions to problems using science as one tool in that endeavor. Like 
engineers, scientists use data to find the truth. They are concerned about reliability 
of the data and what the data reveal about general principles by which we understand 
our universe. Outlier data are often ignored by scientists as noise. Engineers are 
often concerned about outlier data. For example, scientists may be interested in 
better understanding the physics of flight. An engineering team, on the other hand, 
might want to know exactly what outlier conditions 
cause one specific airplane design to crash and for 
what reason. Scalable designs require manufacturing 
tolerances for parts that eliminate outlier failures. 

Margaret Hutchinson Rousseau was the first 
woman to earn a PhD in chemical engineering from 
MIT and the first woman to be accepted as a member 
of the American Institute of Chemical Engineers. 
She oversaw the design of production plants for 
strategically important materials during the war, 
including synthetic rubber and penicillin. 

Pfizer scientist Jasper H. Kane suggested using 
a deep-tank fermentation, a method Pfizer had used 
to produce vitamin C. On September 20, 1942, Pfizer 


purchased the old Rubel Ice Plant on Marcy Avenue FIGURE 0.2. 
Margaret Hutchinson Rousseau 


in Brooklyn, NY.' It had the refrigeration equipment 
Credit: Walter P. Reuther Library, 
Archives of Labor and Urban 
tanks. Dr. Rousseau worked with a multidisciplinary Affairs, Wayne State University 


needed and was equipped with fourteen 7,500-gallon 
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team to turn the Rubel Ice Plant building into the first deep-tank penicillin fermen- 
tation commercial plant. In March 1943, the plant began producing penicillin. 

The process started with a sterile culture of the penicillin mold grown in 
three-liter flasks, then in two-hundred-gallon seed tanks, and finally in one of the 
fourteen 7,500-gallon tanks. Then the drug needed to be extracted from the broth, 
purified, bottled, frozen, and dehydrated. Large quantities of the broth needed to be 
produced because 10,000 parts broth only contained four parts of the drug.” 

By May 1943, deep-tank fermentation produced 400 million units of penicillin. 
In July 1943, the United States War Production Board supported a plan for mass 
distribution of penicillin stocks to Allied troops fighting in Europe. Hutchinson’s 
engineered process and those developed by other engineers such as G. Raymond 
Rettew made it possible for the United States to produce 2.3 million doses in time 
for the invasion of Normandy in the spring of 1944 and over 646 billion units per 
year by June 1945. This engineered process is credited with helping to win the war. 

This story is a victory for both science and engineering. 

To scale innovations for broad use in education and training also requires 
both science and engineering. Learning scientists discover what works in human 
learning, and learning engineering teams develop production models for scaled 
impact. Learning engineering aims to optimize specific learning solutions to under- 
stand under what conditions and with what learners a current design is optimal or 
not, and to develop and test alternative more robust, or 
more refined, solutions that are more scalable. Just like 
with the discovery and production of penicillin, great 
advancements in learning can come from a multidisci- 
plinary approach that recognizes the value of different 
professions and mindsets. 


[2 | 


If you want to learn about learning engineering, a 
good place to start is Pittsburgh, Pennsylvania. What 
was once the steel-making capital of the world is also 
arguably the birthplace of, and a focal point for, the 


emerging field of learning engineering. 
More than fifty years ago, Herb Simon, a 


FIGURE 0.3. World . r 4 : 
War II poster, “Penicillin Carnegie Mellon University Professor, coined the term 


Saves Soldiers’ Lives!” “learning engineering.’* Simon, a Nobel Laureate, 
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was instrumental in launching departments at the 
Pittsburgh-based university that continues to be 
a world leader in cognitive and learning sciences, 
learning technologies, and the application of the 
sciences to optimize human learning. 

Pittsburgh is also home to the Pittsburgh 
Science of Learning Center (a.k.a. LearnLab) man- 
aged by Carnegie Mellon University and the Uni- 
versity of Pittsburgh, and it’s the home of spin-off 
businesses developing some of the world’s most in- 


novative platforms and solutions for learning. 
While the theoretical idea of learning engi- 


mei FIGURE 0.4. 
neering is over fifty years old, the formal definition Herbert A. Simon 


of learning engineering as a professional practice CC-BY-SA by Richard Rappaport 

is still emerging. However, in the pages that follow 

you'll read about teams in Pittsburgh and elsewhere developing exemplary solutions 
using learning engineering practices. Their stories paint a picture of what it means 
to do learning engineering as a professional practice. 


[3] 


In 2018, IEEE—the organization that developed the standards behind WiFi and 
Bluetooth—established the Industry Connections / Industry Consortium on Learning 
Engineering (ICICLE). The group was established to help define and advance the 
profession of learning engineering. Participants in ICICLE include experts from 
fields of learning sciences, instructional design, learning technology, psychometrics 
(an area of study concerned with measurement of knowledge, abilities, attitudes, and 
personality traits), learning analytics, user experience design, and other fields. 

Some of the leaders influencing the work of ICICLE have ties to the roots of 
learning engineering. People like Ken Koedinger at Carnegie Mellon University can 
draw their academic lineage back to Herb Simon, the visionary who coined the term 
learning engineering. With ICICLE, Ken co-led the Curriculum Special Interest 
Group with Mark Lee. That group began the development of a formal definition of 
learning engineering. 

Establishing a clear and concise definition was not easy. Just like today, 
some people felt uncomfortable with the words learning and engineering being 
used together. Others wondered if learning engineering was unique enough, or if 


10 | Learning Engineering Toolkit 


it was just a new label for what they already do. The participants debated what 
makes learning engineering unique based on a first draft by Ken, Mark, and Janet 
Kolodner. Before establishing a concise definition, Ken’s group merged with the 
learning engineering Competencies and Credentials Special Interest Group led 
by Jim Goodell and Michael Jay. The combined groups came to consensus on a 
definition that was adopted by ICICLE. That definition is as follows: 


Learning engineering is a process and practice that... 


1. applies the learning sciences, 
2. using human-centered engineering design methodologies, and 
3. data-informed decision-making 


... to support learners and their development. 


The products of learning engineering are the conditions and experiences under 
which people can achieve great learning outcomes. 

Just like with the discovery and production of penicillin, great advancements 
in learning can come from a multidisciplinary approach that recognizes the value 
of different professions and mindsets. In 2018 and 2019, Janet Kolodner was leading 
ICICLE’s Design for Learning group. Janet is a superstar in the world of the learning 
sciences, having been the founding editor in chief of The Journal of the Learning 
Sciences, a founding executive officer of the International Society of the Learning 
Sciences (ISLS), and a founding program officer of the Cyberlearning and Future 
Learning Technologies program of the National Science Foundation (now called 
RETTL). While leading the ICICLE Design for Learning group, Janet helped launch 
a master’s degree program in learning engineering at Boston College. 

Janet’s group helped develop the idea that learning engineering is often a team 
sport. They recognized the wealth of knowledge and talent in well-developed fields 
of learning experience design. They explored the knowledge, skills, and mindsets 
that were needed for those who design learning experiences and learning systems. 
Then they developed a conceptual model for the kinds of professionals who might 
work with designers to solve learning engineering problems. 

For the 2019 ICICLE Conference, they gave input into a flower diagram with 
petals representing the professional fields that might be part of solving a learning 
engineering problem. Figure 0.5 is a later iteration of that diagram based on input 
from the ICICLE community. 
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In the closing session of that conference, Chris Dede observed that “Learning 
engineering is more of a verb than a noun.” (Chris Dede is the Timothy E. Wirth 
professor in learning technologies at Harvard Graduate School of Education and ed- 
itor of Learning Engineering for Online Education‘ and The 60-Year Curriculum.’ ) 

For some challenges, an individual learning engineer with expertise in multi- 
ple disciplines might be sufficient. But usually, the big problems to be solved through 
learning engineering exceed any one person’s ability. Consider this description of a 
learning engineer from the Educause Learning Initiative’s resource, 7 Things You 
Should Know About Learning Engineering: 


Learning engineers typically have broad knowledge of engineering 
processes as well as of learning science, computer science, and data 
science. In addition to understanding instructional design, artificial 
intelligence and machine learning, pedagogy, and andragogy, they 
offer skills in systems and user experience design, product testing, and 
the development of policies, regulations, and standards. They leverage 
those skills to create learning experiences, courses, and curricula; define 
educational competencies and assessment standards; and use research 
and data to improve teaching and learning.° 


It’s extremely rare for any one person to have expertise in all of these areas. 
Organizations have found it difficult to recruit candidates with such a breadth of 
competencies.’ Indeed, it is probably appropriate to refine the Educause definition 
by making it a definition of a “learning engineering team” as much as that of an 
individual learning engineer. 

Given the multidisciplinary expertise needed on a learning engineering team, 
we envision multiple paths to learning engineering talent development. For example, 
a person may start with a credential in instructional design or software engineering 
and add to that foundation understanding of how people learn. He/she/they may 
add to their knowledge and skills by engaging in formal coursework, learning from 
others on the team, or participating in workshops put on by other professionals. 
Everyone on the learning engineering team doesn’t need to know everything, they 
just need to know enough to communicate with the other experts on the team. (See 
Chapter 10, Tools for Teaming.) 

As important as talent development for the individuals who will contribute to 
a learning engineering team is the development of the team culture, organizational 
processes, and a shared vocabulary. 

We’ve observed learning engineering as a team process that builds on 
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professional specialties within the engineering team as a coordinated effort rooted in 
a shared understanding and vocabulary. Also, we’ve observed that the application of 
learning engineering generally falls into at least three categories: 


1. Learning engineer as consultant (e.g., teaming with an instructor) 


2. Learning engineer as a member of a team designing learning 
experiences, platforms, resources, or other solutions as a contributor or 
as the lead learning engineer that coordinates across the various other 
specialists and sets the direction for the work to be done 


3. Multiple learning engineering professionals—having a base set of 
common competencies and shared vocabularies for the practice of 
learning engineering while having different roles and areas of expertise 


Different professional practices, work processes, and team dynamics apply to 
each kind of team. In teams with multiple learning engineers, the work may be or- 
ganized by smaller role-based teams that influence decisions about different aspects 
of the iterative design. 

While we’ve observed learning engineering being done by teams of people 
who are not called learning engineers, we’ve also observed an emergence of a distinct 
professional discipline and job title. These engineers have specific competencies 
and mindsets and a specific role in the learning engineering process, just as a 
civil engineer has a specific role in the building of a bridge. Chapter 4, Learning 
Engineering is Engineering, covers some perspectives on engineering professions. 
Other chapters include stories in contexts where “learning engineer” is identified as 
a job title. 


[4] 


Located in the culturally diverse East Liberty neighborhood of Pittsburgh’s East End 
is the headquarters for Duolingo, home of the team behind the award-winning lan- 
guage learning platform. It’s no coincidence that the team is located just two miles 
from the Carnegie Mellon University campus where Herb Simon coined the term 
learning engineering. It’s one of several learning technology companies with roots 
in CMU’s Human-Computer Interaction Institute, derived from its Departments of 
Psychology and Computer Science. The company was founded by Carnegie Mellon 
University professor Luis von Ahn and his graduate student Severin Hacker.’ 

On a chilly Friday in November, Mark Lee and Jim Goodell met with Burr 


What is learning engineering? | 13 


Data Software 
Science Engineering 


: x 


Learning 
Learning Experience 
Sciences @ © Design 


N a. 
SHARED iil 
Subject- & UNDERSTANDING © Learning 


Matter Environment 
Expertise Engineering 


al 


Assessment, Education and 
Measurement, Training 
Evaluation Professionals 


FIGURE 0.5. Learning engineering 
teams may require competencies 
from multiple domains 
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Settles, a former postdoc at Carnegie Mel- 
lon University who now leads Duolingo’s 
research group, and his colleague Bozena 
Pajak, a linguist and learning engineer 
who at the time had the title Staff Learn- 
ing Scientist and later became Director of 
Learning and Curriculum. From the out- 
side of the inconspicuous gray building, 
you wouldn’t know that this was the world 


headquarters of the most popular mobile 

FIGURE 0.6. Bozena Pajak and language learning platform, serving over 

Burr Settles at Duolingo 500 million registered users. (Influenced 

by the COVID-19 pandemic, over the 

month of March 2020, new users on Duolingo increased by 101% and learners who 

joined Duolingo during the global pandemic spent 13% more time on Duolingo than 

was typical for new users in the past.?) One of the ways in which Duolingo is in- 

novative is in its use of game mechanics to motivate learners. “One of the tricks 

is figuring out how to marry things that are pedagogically sound with things that 

are reinforcing motivators,” Burr told us. To figure these things out, they use engi- 

neering techniques rooted in scientific research. “We run controlled experiments for 
almost everything we change,” he said.'° 

This kind of testing and iterative improvement is a common practice among 
practitioners of learning engineering whether the product is an app, course content, 
or changes to a learning environment. 

For every design decision that the Duolingo team makes, they have to consider 
multiple metrics of success. Those metrics include measures of learning, measures 
of user engagement, and how user interactions impact the company’s bottom line. 
They know that to continue to provide free language learning to millions of people 
around the world they need to stay in business. This means that every decision is a 
balance between these metrics. If they can improve one metric without hurting the 
others, they do it. 

The team at Duolingo improves its products through its own scientific re- 
search while applying the scientific discoveries of others. For example, they know 
from prior learning sciences research that when learning new words, the best time to 
practice is when you are on the verge of forgetting. 

“Our approach is based on the spacing effect: the finding that short practices 
spaced out over time is better for learning than ‘cramming.’ A related finding is the 
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lag effect: You can learn even better if you gradually increase the spacing between 
practices. These ideas go back to 1885, when German psychologist Hermann 
Ebbinghaus pioneered the concept of the forgetting curve.” |! 

The science of learning is informed by what we know about how the mind 
works. These sciences include: !? 


e Neuroscience: Biological studies of the nervous system and brain 

e Cognitive psychology: Study of the human mind through behaviors 

e Learning sciences: Study of learning in realistic settings 

e Education research: Models and interventions at the classroom+ level 

e And others: As well as many other disciplines, such as developmental 
psychology, social psychology, and anthropology "° 


Learning engineering is informed by the combination of what each of these 
research traditions has to offer. Learning engineering also can benefit from research 
in areas of cognitive and social psychology such as behavioral economics (the 
study of psychology as it relates to economic decision-making) that can help us 
in understanding learner motivation and engagement. Emerging opportunities in 
learning engineering take advantage of biomedical research and development for in- 
novations, such as devices that monitor learner attention levels using EEG (electro- 
encephalogram) sensors that monitor brain waves. Finally, learning engineering 
is increasingly the application of data science—an interdisciplinary field that uses 
scientific methods, processes, algorithms and systems to extract knowledge and in- 
sights from data. It has been called the “fourth paradigm for scientific exploration.” 

Some of these sciences are relatively new. Neuroscience was first recognized 
as a unique academic discipline in the 1950s and 1960s," cognitive science was 
established in the mid-1970s,!° and learning sciences emerged in the 1990s.” And 
yet the pace of discovery has accelerated with the help of innovations like functional 
magnetic resonance imaging (fMRI) and genomic mapping of the brain.'® 

Learning science findings applicable to learning engineering include cog- 
nitive, noncognitive, and motivational aspects of learning. Environmental and 
noncognitive physiological factors are also considered, such as physical health and 
safety, relationships, and social contexts. 

Learning engineers are concerned not only with how the brain works in 
general, for the average person, but also understanding and empathizing with the 
uniqueness of each learner. In The End of Average, Harvard Professor Todd Rose 
showed that no one is an “average size.” The size of a person can be measured on 
multiple dimensions (height, weight, shoulders, reach, chest, torso, waist, hips, legs). 
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To understand a person’s learning and talent development requires measuring many 
more dimensions than those physical attributes of size. 


Just about any meaningful human characteristic—especially talent— 
consists of multiple dimensions. The problem is that when trying to 
measure talent, we frequently resort to the average, reducing our jag- 
ged talent to a single dimension like the score on a standardized test or 
grades or a job performance ranking. But when we succumb to this kind 
of one-dimensional thinking, we end up in deep trouble.!” 


Two students could get the exact same set of questions wrong on a test, but for 
different reasons. Promoting robust learning requires addressing those multiple rea- 
sons. Learning engineers ask the why questions to determine the multi-dimensional 
reasons that individuals might or might not master a learning objective under various 
conditions. With insights into possible root-causes, they create and test solutions that 
might overcome those problems. 

Good learning engineers have a deep understanding of how the human brain 
learns and develops: “Because trying to reengineer learning without being versed in 
the basics of learning is like trying to improve brain surgery without knowing much 
about the brain.” *° 


[5] 


An engineering design methodology particularly suited to learning engineering is 
called huma 


i : n, a problem-solving process that begins with empa- 
thetically understanding a challenge that people face and then iteratively developing 


and discovering a best fit solution for a given person or group to meet a specific need 
within a specific context. 

Human-centered design existed as a practice long before it was formalized 
and given a name. In 1894, Will Kellogg used human-centered design to invent a 
breakfast cereal.”! His older brother John Harvey Kellogg was a medical doctor and 
director of a hospital in Battle Creek, Michigan. Will, a former broom salesman, 
moved to Battle Creek to help his brother at the hospital. They had empathy for the 
patients who were struggling to eat toast in the hospital setting. John was a nutrition- 
ist and concerned about the digestibility of his patient’s foods. They wanted to offer 
an alternative. With this need in mind, Will experimented with a pot of boiled wheat 
that he had left out overnight. He rolled it out and baked it. The crispy flakes were 
the first of many iterations over several years to perfect the process. He gave it to 
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FIGURE 0.7. Sciences of learning are a foundation for learning engineering 


CC-BY Jim Goodell, 2021 


his brother’s patients, confirmed the flakes were easier to eat, and got their feedback 
with each iteration. He eventually used corn and invented corn flakes. 

Today human-centered design is used to better match product designs to the 
needs of the people who will use them. The process is being used to design every- 
thing from software user experiences to entire school districts.” The process has 
been formalized as an international standard (ISO 9241-210) for developing interac- 
tive technology, but the concepts are the same even when applied to non-technical 
problems. With user-centered design: ** 


Designs use an explicit understanding of users, tasks, and environments 
Users are involved throughout design and development 

The design is driven and refined by user-centered evaluation 

The process is iterative 

The design addresses the whole user experience 

The design team includes multidisciplinary skills and perspectives 


NwAPWwWN EP 


In the case of learning engineering and instructional design, the human- 
centered design challenge is for a set of people to learn something. Principles 
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and practices of engineering help further define how the work of developing and 
improving learning solutions gets done. Learning engineering combines design 
methodologies with data-driven processes and principles of engineering—such as 
engineering control theory—not typically found in traditional design practices, to 
iterate closer and closer to a best fit solution. 

Deeply understanding the people who will use or benefit from the solution is 
an important idea, whether that solution is a piece of software or something lower 
tech, like a breakfast cereal or, as we discuss below, a new kind of donkey cart. 


Learning engineering is more of a verb than a noun. 


- Chris Dede 


[6| 


The Duolingo team develops software, but technology isn’t the only product of 
learning engineering. It could be a low-tech learning resource, an improved process, 
a physical environment; anything designed to provide learning experiences and 
conditions for learning that work; anything developed to solve a learning problem. 
Technology is just one of the tools in the learning engineering toolkit. Sometimes 
the needed solution is low tech. 

Shelly Blake-Plock was the founding chair of ICICLE. Shelly is a technical 
genius and the founder and CEO of Yet Analytics, a company that specializes in data 
analysis in support of human learning. But he will be the first to tell you that learn- 
ing engineering is not about the technology, it’s about problem solving in support of 
learning. He saw this firsthand when he visited the West African country of Gambia. 

The problem to be solved was student access to local schools. “In a concerted 
effort to reduce inequity, the government abolished school fees and increased the 
number of communities that have schools within three kilometers from 83 percent 
in 2013 to 95 percent in 2018,” according to a story posted to the World Bank web- 
site.” However, nearly 30 percent of school-aged children still didn’t attend classes. 
A problem was that a three kilometer walk to school was too far for three- to eight- 
year-old students. Parents were concerned about their children’s safety over the long 
distances. 
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The problem was framed by the context of poorly developed or nonexistent 
roads. “Donkeys, a common means of transport in The Gambia, were a natural, yet 
innovative, choice,” according to the World Bank article, “and different communities 
across the country were enthusiastic about the idea.” Donkey carts were engineered 
to carry ten young children each and could be driven by an older student. 

A key strategy in any engineering design is iterative improvement. The low- 
tech donkey cart solution used an iterative process. “An initial pilot revealed that 
while the original carts were an effective means to transport children to school, 
these were too heavy and needed to be redesigned. As a result, a lighter 2.0 model 
of the cart was launched in 2015. Built with locally sourced materials, the revamped 
carts handle more easily, have a seat for each child, and incorporate important safety 
features such as a gate and easy-to-grab handles.” For more on learning engineering 
that’s helping people in underdeveloped countries see Chapter 3 for a story about a 
team developing a mobile app for people in another West African country, Ghana. 

When Herb Simon first coined the term learning engineering over a half 
century ago, the tools of the trade available now didn’t yet exist. Key learning 
sciences concepts had yet to be discovered. The most advanced learning technology 
outside of academia might have been the rudimentary screen-less flight simulators 
being used by the military and commercial airlines to train pilots.” For academic 
classrooms, state-of-the-art technology was the chalkboard, the textbook, overhead 
projectors, and the mimeograph machine. 

Today, teams of learning engineers have rich sets of learning sciences and 
technology tools that can be applied toward realization of Herb Simon’s vision for 
learning engineering. These tools include ubiquitous internet-connected devices, 
sophisticated software substrates and development tools, artificial intelligence, 
virtual and augmented reality, project-based and collaborative pedagogies, and 
online and blended learning modalities. (See Chapter 14, Software and Technology 
Standards as Tools.) 

Among the most powerful tools in the learning engineer’s toolbox is the abil- 
ity to collect and use data to iteratively improve the learning design and iteratively 
improve the solution for any given learner. 


[7 | 


In 1998, Carnegie Mellon University spun off Carnegie Learning, Inc. to support 
scaled adoption of cognitive tutors, the kind of intelligent tutoring system developed 
through the university’s research. Carnegie Learning’s MATHia offers one-on-one 
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tutoring in mathematics covering content for grades 6, 7, 8, Algebra I, Geometry, 
and Algebra II. The intelligent tutoring software presents problems for students to 
solve; it provides feedback to students as they are working through problems based 
on answers given and other data. According to their website, the software “monitors 
the status of the student’s knowledge on a moment-by-moment basis and tailors 
course material, based on these continual assessments.” The twenty-two-year-old 
company merged with New Mountain Learning in 2018. 

While Carnegie Learning doesn’t use the label “learning engineer,” they’re 
doing learning engineering as a “team sport,” bringing together groups of specialists 
to solve problems. A cognitive science team, a data and research team, an instruc- 
tional design team and everyone developing and maintaining the product as a whole 
might be called the “learning engineering team.” They often bring together people 
from the various groups to have “data jams.” When the data scientists discover a 
case where the data are telling them that something is not working for a subset of 
students, they call a team meeting in which everyone works together to find the root 
cause of the problem and suggest possible solutions. 

Groups that participate in “data jams” don’t all need to understand data sci- 
ence, but “they certainly need to basically understand how to think about data,” 
Steve Ritter told Jodi Lis, Mark Lee, and Jim Goodell when they visited Carnegie 
Learning’s Pittsburgh headquarters in 2018. 

Ritter’s team builds tools to support the non-data-scientists on the team. “They 
use the tools to try to figure out how to improve instruction, and we use the tools to 
figure out what this data means,” he said, “is there a better way to sort the world, to 
figure out what’s really going on?” 

Michael Sandbothe, the lead programmer on data tools at the time, worked 
to make it so the end users need the least amount of high-level data science skills. 
“You don’t need a PhD in statistics to use the tool,” he said. Stephen Fancsali, a 
research scientist on the team, might find an inconsistency in the data but does not 
know what’s happening. Why are students able to answer one mathematics prob- 
lem but not able to answer one just like it? The instructional design teams, former 
mathematics teachers, and other content experts have insights that the data scientist 
might not have. They might say, “Here’s what’s really happening, the students are 
confused about this.” These kinds of insights become hypotheses that the research 
team can test against the data. More insights from Carnegie Learning and others on 
data-informed decisions are available in Chapter 6. 

Continuously examining the data to discover how to make a product or solu- 
tion better for learning is a key part of the learning engineering process. The method 
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for deciding what data to collect and how it will be collected is called “instrumenta- 
tion,” and is covered in Chapter 5. 


[8| 


If you use Duolingo to learn a new language, the app can predict your future. After 
you’ve been introduced to a new word and had some practice with it, the app can pre- 
dict whether or not you will have forgotten it at some point in the future. To predict 
the future, it doesn’t use a crystal ball, it uses data and a scientific discovery made 
more than a century ago. 

In 1879, German psychologist Hermann Ebbinghaus was curious about how 
memory worked. He created experiments to test how well he could recall new words. 
He knew that he would be influenced by prior knowledge if he used real words, so 
he created a list of nonsense syllables composed of three letters (a consonant on each 
end and a vowel in the middle). He eliminated syllables that sounded like or were 
parts of real words (for example, BOL or PED, respectively). Then he tested himself 
by randomly looking at a sequence of cards with the nonsense syllables, writing 
them in a notebook, and then trying to recall the list.*° 

In 1885, he published findings about the learning curve, how fast people learn 
new facts, and the forgetting curve, how fast people forget what they’ve learned.” 
His discoveries about the forgetting curve are the basis of Duolingo’s ability to pre- 
dict your future. 

Additional research over the last century has led to a more complete scientific 


understanding about the spacing effect,” the finding that people are more likely to 
remember something if they par out the learning over time rather than cramming 
by trying to learn it all at once. In the case of learning a new language, newly intro- 
duced and more difficult words should be reviewed more frequently, and older or less 
difficult words should be reviewed less frequently. 

Another phenomenon, the lag effect, is that people commit information to 
long-term memory better if the time between study or practice increases gradually. 


Yet other research showed that the best time for people to practice is right before 


they’re about to forget something. Spaced repetition models try to figure out when 
the best times are to review or practice specific facts or skills.” 

Duolingo’s Burr Settles and a former Duolingo employee, Brendan Meeder, 
who later moved to Uber’s Advanced Technologies Group, developed a “trainable 
spaced repetition model.” It’s software that uses data and artificial intelligence to 
learn the best time to have people practice specific words in a new language that 
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they’re learning. This model used by Duolingo isn’t the first of the so-called for- 
getting functions, but it’s a new approach that uses modern machine learning tech- 
niques. It takes in data about a student’s practice history (when they practiced and 
whether or not they remembered correctly) and data that helps the machine learning 
algorithm know how difficult a word is to memorize compared to other words. More 
insights from Duolingo are found throughout this book. 

Like Margaret Hutchinson Rousseau and the engineering team at Pfizer, they 
took a scientific discovery and engineered a solution that helps hundreds of millions 
of people. Using data and a century-old scientific discovery, Duolingo can person- 
alize and optimize your learning of a new language—by predicting your future, for 
example, knowing when you're likely to forget a word. x 


Learning engineers and learning engineering teams apply the learning 
sciences using engineering design processes and data-informed 
decision-making to create the conditions for optimal learning. 


Engineering is different from science. Learning scientists discover what 
works in human learning. Learning engineers aim to optimize particular 
learning experiences, test different options, and build scalable solutions. 


Learning engineering is often a team process that brings together 
professionals from different fields, including the learning sciences, 

assessment, learning experience design, software engineering, and data 
science. Team members don’t need to be experts in all areas. 


Technology is a tool that can help scale the implementation of learning 
experiences. It also enables data-informed decision-making. However, 
learning solutions don’t necessarily require advanced technology. 


Learning engineers design learning experiences, but that’s not all they 
do. They also address the contexts and conditions that lead to great 
learning. These might include the architecture of physical or virtual 
learning environments, social structures, learners’ mindsets, and habits 
of practice as well as more obvious targets, such as curriculum design, 
educational technology development, and learning analytics. 


Learning engineers don’t design for the “average” student; they engineer 
adaptations and variations to fit real people and sometimes even 
individual learners’ needs. 


Data instrumentation and analysis are critical to learning engineering. 
They inform an iterative process of creation, implementation, and 

investigation. Data can help optimize learning activities and feedback, 
such as Duolingo’s trainable spaced repetition model. 


Learning engineers keep in mind the science of how people learn and 
how people are motivated to learn. 


Learning engineering takes a systemic view of learners’ broad 
experiences, including the life experiences that impact their learning. 
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